A Comparative Analysis of Thermal and Visual Modalities for Automated Facial Expression Recognition

Facial expressions are formed through complicated muscular actions and can be taxonomized using the Facial Action Coding System (FACS). FACS breaks down human facial expressions into discreet action units (AUs) and often combines them together to form more elaborate expressions. In this paper, we present a comparative analysis of performance of automated facial expression recognition from thermal facial videos, visual facial videos, and their fusion. The feature extraction process consists of first placing regions of interest (ROIs) at 13 fiducial regions on the face that are critical for evaluating all action units, then extracting mean value in each of the ROIs, and finally applying principal component analysis (PCA) to extract the deviation from neutral expression at each of the corresponding ROIs. To classify facial expressions, we train a feed-forward multilayer perceptron with the standard deviation expression profiles obtained from the feature extraction stage. Our experimental results depicts that the thermal imaging modality outperforms visual modality, and hence overcomes some of the shortcomings usually noticed in the visual domain due to illumination and skin complexion variations. We have also shown that the decision level fusion of thermal and visual expression classification algorithms gives better results than either of the individual modalities.